5 Construct Validity and Cognitive Diagnostic Assessment

Reasoning Test The ART was developed by Embretson (1995) using the cognitive design system approach. ART items are matrix completion problems. Matrix completion problems appear on a variety of intelligence and ability tests. They are considered a stable measure of fluid intelligence (Raven, 1965). Construct Validity and Cognitive Diagnostic Assessment 133 Embretson (1995, 1998) further developed the processing theory in two ways, that is, combining the relational processing variables into a single variable, namely, memory load, and including an encoding stage that is influenced by several perceptual features of items. The memory load variable includes both the number and the level of relationships. Carpenter et al. (1990) hypothesized that individuals attempted to solve the item with the lowest-order relationships before attempting higherorder relationships. Accordingly, for the highest-level relationship (i.e., distribution of two), all lower-order relationships are assumed to be attempted. The memory load variable is a count of the total number of relationships attempted before reaching the required relationships in the problem. Encoding is influenced by the number of unique attributes, degree of stimulus integration, object distortion, and object fusion. In matrix completion problems, more than one object may appear in a single cell of the design. Number of unique attributes refers to the number of separately manipulated objects in the problem stem. Stimulus integration refers to the arrangement of the objects. The most integrated display occurs when objects are overlaid, while the least integrated display occurs when two or more objects are displayed around a platform, such as a "-". Object distortion refers to corresponding objects for which the shape of one or more is distorted. Finally, object fusion occurs when overlaid objects no longer have separate borders. Estimates of item difficulty from the one-parameter logistic item response theory (IRT) model for 150 ART items were available. The estimates were based on a large sample of young adults. For this set of items, five different items had been generated from each of 30 different item structures. The five variant items differed in the objects or display features. Scores for the items on all cognitive variables were available. Estimates of item difficulty were regressed on the cognitive variables scored for each item. Table 5.1 presents the overall model summary. Two TABLE 5.1. Regression of item: difficulty on cognitive model variables Model R R Adjusted Std. error square R square estimate Change statistics R square change F Sig. F change dfl df2 change 1 Structural . 758 .575 .569 .92893 .575 99.475 2 147 .000 only 2 Structural .782 .612 .598 .89737 .036 4.508 3 144 .005 perceptual 132 Xiangdong Yang amd Susan E. Embretson